Automated functional understanding and optimization of human/machine systems

ABSTRACT

A method of analysing and tracking machine systems has the steps of sensing operational data from equipment, the operational data comprising at least location, time, and one or more operational condition data related to the equipment; analysing the operational data to identify data patterns; logging the data patterns as events in a database; comparing the events to a database of predetermined patterns to classify each data pattern as a known event or an unknown event; updating the database to include a new data pattern related to any unknown events; and alerting a user to further classify the unknown events manually.

FIELD

This relates to methods and systems for analysing and tracking entitiesoperating as part of an organization or system involving both humans andmachines.

BACKGROUND

In order to coordinate groups of humans, tools, machines, and resources“operations management practices” are used. These practices rely ondecision making based on available information and relate to actions ofbenefit to a group of humans (e.g. a corporation, governmental group,non-profit society, religious sect, etc.). United States pregrant pub.no. 20040210371 (Adachi et al) entitled “Location confirmation methodand location display system of construction machine and suchconstruction machine” and United States pregrant pub. no. 20140310412(Shinohara et al.) entitled “Management server for remote monitoringsystem” describe systems for tracking the operation and movement ofindustrial equipment.

SUMMARY

There is provided methods and systems for automatically sensing,abstracting, perceiving, and classifying the actions and identities ofentities operating as part of an organization or system involving bothhumans and machines. The system automatically senses and classifiesentity actions and, as well, partitions them into time/space regions inwhich they occur. Such information can be used to analyse and partitionsystem activities, allowing identification and understanding ofclassified groups of actions and how they relate to organizationaltactical and strategic goals. Based on this action/region partitioning,computer models for classes of entities within an organization may bedeveloped, inferring causal rules for how each entity responds tochanges in their sensed state. Based on such entity modelling,predictive simulations may be constructed to assess the probable effectof operational changes on the system as a whole. Based on suchinformation, optimization of organizational functioning can be achieved.Optimization can be both local (e.g. based on, for example, a single,particular, entity) and global (optimization of an entire, coordinatednetwork of system entities).

According to other aspects, there is provided methods and systems forautomatically perceiving, analysing, and reporting on actions of one ormore entities and/or associated entities within an organizationcomprised of humans and machinery, comprising: Geographic position andtime measurement; At least one sensor detector; Electronic transductionto convert sensor signals and geography/time information into computernetwork transmissible form; A time/space-ordered database to record saidtime/space/sensor measurements; Computational device(s) to automaticallyabstract, recognize, and classify at least one set of actions from saiddatabase, thereby determining the time/space “Region” of extent of theactions; A “markup” database capable of storing such perceived actionsand their metadata; Action analytics and reporting device(s) capable ofdetermining facts about at least one entity, storing such facts in adatabase for future reference/comparison, analysing such facts withrespect to previous historic facts and actions, generating exceptionalerts regarding significant deviations from either historic averages ofactions or absolute thresholds, generating profit/loss/cost accountingsummaries for at least one entity, and generating automatic billinginformation based on entity actions.

According to other aspects, there is provided methods and systems forautomatically measuring, abstracting, analyzing, categorizing, andunderstanding—in real-time or near-real-time—actions (both in time andspace) of organizational entities. Such automated, data-driven,perception of entity actions enables automatic summaries and time-motionproductivity analysis of entity activities over arbitrary andnon-continuous blocks of time and 3D space. It also allows suchsummaries to automatically adapt over time, changing in response tochanges in the nature of the activities being monitored. As well, thesame abstracted data can be used to automatically build computationalmodels of organizational entities. These models that can be used foroptimization purposes—either optimizing actions of the entity inquestion, or linking such models together with implicit data-definedinter-entity relationships to optimize groups of entities on anorganization-wide basis.

According to other aspects, there is provided systems and methods thatmay be used to create a networked, geographically distributed, entitysensing network capable of real-time or near-real-time perception andanalysis of the states and actions of many entities at once. For eachentity, this consists of: acquiring real-time data from a multiplicityof sensors, amongst which are some allowing GPS geographic locationmeasurement; storing such geo-location data in a time-ordered 4dimensional (three spatial dimensions plus time) database ofobservations that effectively establish what in physics would be termedthe entity's “world line”; pre-processing this data using signalprocessing techniques to increase signal-to-noise ratio and, as well,help recognize patterns within and between signals; analysing this worldline data to recognize and classify multiple tiered levels of abstractedactions, or data structures, termed “Gestures”, “Behaviours”,“Activities”, and “Accomplishments”; automatically linking suchidentified actions with particular “Regions” of space/time within whichthey occur; entering all such information into a “world line markup”database overlaying the entity's 4D world line; using this informationin combination with raw observations to ascertain the entity's identitybased on a “fingerprint” of its real-time actions; analysing theentity's marked up world line to tabulate statistics regarding thenature and duration of all perceived actions; generating operationalsummaries thereof, along with real-time exception events (deviationsfrom absolute thresholds or historical averages), per entityprofit/loss/cost accounting summaries; and performing automated billingbased on entity actions.

According to other aspects, the systems and methods may utilize machinelearning, pattern recognition, and other computational techniques toautomatically and inductively infer/identify important variables andrelationships based on comparison between “constellations” of currentdata with the ongoing historic dataset already acquired overdays/weeks/months/years of operations. Such variables/relationships areextracted from historic data as it is acquired and are then used toformulate predictions and hypotheses about future states of the system.As the existence and predictive utility of such variables/relationshipschanges over time, this system and method perceives its environment,learns, and adapts, changing its models to accurately anticipate thefuture in real or near-real time, and allowing for optimizing oforganizational behaviour accordingly.

According to other aspects, through automated real-time andnear-real-time perception, analysis, categorization, and understandingof entity actions, the system enables multi-dimensional,operations-context/application-specific, optimization within anorganization. It enables both sophisticated localized (single entity)optimization and, as well, global enterprise-wide optimization forspecific high level goals such as profit, business growth, et al. Itenables end-to-end, dynamic real-time or near-real-time, optimization oforganizational functioning in terms of overarching goals such asprofitability (per entity, or per group of entities), pickup/deliveryefficiency, or business growth.

In other aspects, using the system, it is possible to observe singleentity or grouped entity response in near-real or real-time asorganizational changes are implemented, assessing the system's overallresponse to the changes implemented and comparing it with otherexperimental changes made in the past or predicted. Suchsystems/networks consist of one or more organizational entitiestraversing a “world” (i.e. the physical world and/or one or more virtualcomputer simulation(s)) in both time and space with the goal ofperforming optimal actions at many different locations, such locationschanging over time.

According to other aspects, the systems and methods automatically andcontinuously sense entity state through a multi-dimensional array ofsensed/measured variables in combination with a variety of contextualenvironmental variables. Data from each sensor are serialized and storedto form a series of 4D (3D plus time) “world line” measurements. Suchworld lines effectively arrange the sensor data in time/space, tracingout the movement of the entity through these 4 dimensions. As part ofthe action abstraction and perception process, “Constellations” oftime-coherent world line measurements are built, copying andcross-linking relevant data from multiple sensors to form constellationsof readings that all occurred within particular bounded temporalregions. Constellation members are chosen to reinforce one another toheighten their statistical certainty as indicators determiningstart/stop of useful events. Such world lines are stored in a special,temporally ordered, database.

In some aspects, based on the observational world line data flux fromsuch sensing, the system scans the multi-dimensional data flow from eachentity, automatically identifying operations-specific “Observations”encountered/generated by the entities. A combination of signalprocessing and machine learning algorithms examines thismulti-dimensional data, identifying high probability transition pointswhere activity shifted from one endeavour to another, and, as well,identifying when constellation values fall within “definitions”identifying and classifying particular actions. Such definitions can beeither explicitly defined by human programmers, or automaticallyinferred through so-called “unsupervised” signal processing and/or datadiscretization techniques known to one skilled in the art. The systemthen links these Observations to time-based geographic locations,abstracts other, higher level events by using data fusion techniques toanalyse temporally adjacent lower level or same level eventcombinations, links sequences/patterns of events into time-based“gestures”, associates gesture sequences/patterns into “Behaviours”,automatically determines 4D world line “Regions” of geographic/temporalextent within which such Behaviours occur, and automatically overlaysthe world line of each entity's physical/temporal trajectory with aseries of segmented Regions extending through time and space(“auto-segmentation”). The system then integrates the totality of suchautomatic abstractions to identify, measure, and tally completion of“Activities” of interest that are further combined/abstracted to form“Accomplishments” that form the granular bedrock of tactical andstrategic operational goals.

According to some aspects, the automatically machine-perceivedinformation as discussed above may be used to multi-dimensionallyoptimize performance, allowing the system to automatically perceive andadjust (or be adjusted by human operators) to variations in both thephysical world and entities' states over time so as to optimize theefficiency and profitability of the system/network. Such optimizationscan happen automatically, or in a human assisted, interactive fashion,adaptively optimizing system/network behaviour.

According to some aspects, the block functionality of such systemcomprises: acquiring data from a variety of real-time and near-real-timesensors and, if necessary, pre-processing it using local “embedded”computing resources; transmitting that multi-sensor data to amulti-appliance computing platform (generally cloud based) that performsadditional computing/storage tasks; serializing the data flow so as toensure proper temporal ordering of individual sensor data; constructingtime-stamped 4D “world line” data streams for each sensor that arestored in, for example, a NoSQL database; cross-linking said world linedata to create “constellations” of intertwined world line data thatcombine multiple sensor measurements in ways useful to determining eventstart/stop boundaries and associated measurements; abstracting“Observation” features through analysis of constellation variables usingone or more layers of computational processing and optionally storingsaid abstracted features into one or more separate database(s);abstracting “Gestures” from such streams of temporal events, recognizingthem as repeating patterns of Observations and other Gestures extendingover time and space; combining time sequenced sets of Gestures torecognize “Behaviours” as time/space patterns of Observations, Gestures,and other Behaviours; recognizing “Activities” as time/space patterns ofObservations, Gestures, Behaviours, and other Activities; andrecognizing “Accomplishments” as time/space patterns of Observations,Gestures, Behaviours, Activities, and other Accomplishments;automatically measuring and associating these Gestures, Behaviours,Activities, and Accomplishments with Regions of time/space extent;entering all such information into a “world line markup” databaseoverlaying the entity's 4D world line; using this information incombination with raw world line Observations to ascertain the entity'sidentity (and that of its human operator, an “associated entity”) basedon a “fingerprint” of its real-time actions in comparison with priorhistory; further processing said features, creating record entries in andatabase of entity facts; using such information to analyse and reporton entity performance in terms of operational summaries tabulatingGestures, Behaviours, Activities, Accomplishments and metricssurrounding their execution, exception events (deviations beyond eitherabsolute thresholds and/or historic performance averages of certaintasks), profit/loss/cost accounting tabulations; and automaticallygenerating billings based on entity actions; to feed the combined entityworld line observations, action/region markup, and measured analyticfacts to a Historic Modelling software module that uses such data toinfer causal relationships between variables and create a software modelof the entity and how it responds to changing variables over time.

According to some aspects, the method and apparatus discussed hereinincludes a properly integrated combination of the following elements:

-   -   a) “Geographically Indifferent Data Acquisition” whereby        real-time or near-real-time measurements from each entity's        sensors are acquired, locally pre-processed or        conditioned/scaled, and transmitted over a wired or wireless        electronic computer-based network to a special purpose database        server or server network.    -   b) “Serialization and World Line Creation” whereby the incoming        raw data is time-ordered, and inserted into a custom structured        database containing sensor readings accessible via the 3 spatial        dimensions plus time, such a database being meant to function        both as a “big data” repository for analysis of contemporaneous        data, and, as well, a historic “memory” of past activities.    -   c) “Time and/or Spatial Signal Processing” whereby sensor data        streams, either alone or in combined constellations are        processed using signal processing techniques meant to extract        information regarding the structure of the data flow such as,        for example, periodicity, frequency spectra, self-similarity,        wavelet basis set composition, et al. Such processing may be        performed upon time domain and/or spatial domain (concerning        geographic 3D location(s)) data as appropriate. Additionally,        such streams may be mapped from non-temporally adjacent windows        of time to form new composite data streams containing multiple        data streams time-offset and/or space-offset from one another.    -   d) “Gesture, Behaviour, Activity, and Achievement Recognition        and Classification”, or types of data structures, wherein both        raw and time/space processed sensor signals are analyzed and        compared to extant definitions of both actions and geographic        spatial regions to identify recognizable action patterns of        aggregate raw and abstracted variables, and identify when/where        each action starts/stops in both time and space. Analysis        further recognizes/classifies them (if known) by type, and, if        unknown, automatically develops definitions for them, tags them        as new unique types, and flags them for metadata entry by human        operators. As the data structures progress from gesture through        to activities and beyond, the type of data structure may be        considered to fall within a hierarchical order from the lowest        order, such as data patterns, to higher order data structures.    -   e) “World Line Time/Space/Region Auto-Segmentation” wherein        entity world lines are automatically segmented to create a world        line markup database that overlays each entity's world line with        identified actions and the 4D (space plus time) regions in which        said actions occur.    -   f) “Action-Based Analytics, Reporting, and Billing” in which the        segmented 4D world lines of entities are analyzed, summarized,        and compared with historic performance of the same, different,        or aggregate-averaged entities' performance. Said analysis        produces operations summaries and comparisons, exception events        (where actions are problematic and/or deviate significantly from        historic practice), and profit/loss/cost accounting summaries on        a per entity basis that can also be aggregated across either        groups of entities or an entire organization. Further, the        evidence-based reporting of automatically detected and measured        actions can be used to create automated billings based on actual        events that occurred, not contractual generalities, enabling        near-real-time evaluation of individual entity profit/loss/cost        and response to changing entity or system conditions over time.

According to some aspects, the system may include one or more of thefollowing features:

-   -   A system that stores sensor data linked to 4D (time plus 3        spatial dimensions) world line locations, uses such time/space        positioning to analyse sensor signals, abstract patterns from        them that identify actions, recognize, perceive, and classify        multiple tiered levels of actions and relationships between        them;    -   Automatic detection and classification of entity actions, and        measurement of their Regions of extent over both time and space;    -   Automatic derivation of the metadata structure of relationships        between entity actions, resulting in a tiered perception of        levels of actions—components that build on each other to allow        perception of larger, more comprehensive, activities and        accomplishments;    -   Automatic identification of potential cause/effect relationships        within said metadata structure of relationships;    -   Automatic analysis and reporting on entity actions to establish        per entity profit/loss/cost profiles;    -   Automatic reporting of exception events regarding automatically        detected deviations from normal historic averages for either the        entity per se or aggregate averages across a truck fleet,        different drivers, etc.;    -   Automatic analysis and reporting across entities to establish        automatically calculated and continuously updated aggregate and        entity-type-specific normative averages and variances for one or        more groups of similar entities;    -   Use of continuously updated normative figures to create a        continuously adaptive method of perceiving and identifying        outlier actions based on deviations from norms that change over        time (essentially a way of auto-thresholding detection of        actions of note);    -   Ability to automatically adapt reporting of exception and/or        outlier events over time based on deviations from continuously        updated normative averages and variances;    -   Automatic analysis of, and continuous updating of, per entity        and cross entity efficiency measurements;    -   Custom Gesture/Behaviour/Activity/Accomplishment variable        windowing of data flows based on definitions that are either a        priori from humans, or derived by machine learning pattern        analysis (i.e. the specific Gesture detection techniques        described);    -   Jointly/Severally doing signal processing of sensor signals        based on any combination of time plus 3 spatial dimensions plus        the sensor values themselves;    -   Signal processing Time and/or Geographic Window Assembly        subsystem functionality for compositing signals of known time or        geographic offset;    -   Identifying each entity action by a particular “fingerprint” of        sensor data flux over particular regions of time and space.        Transitions between these can be automatically recognized using        machine learning data discretization techniques. This allows for        a dual automated recognition of transition boundaries followed        by automated derivation/definition of indicators for identifying        the particular action (based on the data flux “fingerprint”        interior to the start/stop transitions);    -   Tiered abstraction and perception/recognition of successive        levels of actions, each built upon a combination of raw        Observations and previously perceived/recognized, lower level or        current level actions, such perception/recognition being based        on multi-dimensional matching of either human-defined or        machine-generated action definitions (e.g. Gestures);    -   Identify and classify both entities and “associated entities”        automatically through machine learning techniques examining        their sensor data fluxes, perceived actions, and time/space        relationships between said actions. Each entity such as (in a        preferred embodiment) a truck has a particular “fingerprint”.        This fingerprint is affected by the associated entity (e.g. the        driver of a truck). Transitions between these can be recognized        using machine learning data discretization techniques. Having        segmented such transitions, the body of each separate entities        data flux can then be analysed for maximum likelihood indicators        that then can be automatically set as definitions to identify an        entity's and/or their associated entity's presence during        particular time periods.    -   Overall entity performance analysis may be based on cyclical        temporal analysis and/or signal processing techniques to        identify patterns in the performance data set and deviations        from historic norms.    -   Overall entity performance analysis may be based on machine        learning algorithm approaches similar to those already detailed        for entity action recognition/classification, allowing automatic        segmenting/classification of entity performance, development of        maximum likelihood estimators to identify each classification        type, and analysis/establishment of cause/effect relationships        between variables. This automatic elucidation of the structure        of each entity's performance and creation of cause/effect        understanding of the causes of such structure is a significant        advance over present day organizational analysis capabilities.    -   Use of machine learning and other techniques to automatically        detect and elucidate the structure of actions of entities and/or        associated entities in the system. This applies not only to the        actions, but the relations between the actions.    -   Automatic detection of performance deviations from historic        functioning    -   Automated action-based billing    -   Assessment of performance response to a known recipe of        operations changes

According to an aspect, there is provided a method of analysing andtracking machine systems, comprising the steps of: sensing operationaldata from equipment, the operational data comprising at least location,time, and one or more operational condition data related to theequipment; analysing the operational data to identify data patterns;logging the data patterns in a database; identifying one or moregestures by comparing the data patterns to a set of gesture definitions;and identifying one or more behaviours in a set of behaviourdefinitions, each behaviour definition comprising a gesture and one ormore of: one or more additional gesture, one or more operational datum,or combination thereof.

In other aspects, the method described above may further comprise thefollowing aspects, alone or in combination: the method may furthercomprise the step of monitoring for unknown gestures or unknownbehaviours, and automatically adding a definition of the unknowngestures or unknown behaviours to the respective set of gesturedefinitions or set of behaviour definitions; a user may be alerted toclassify unknown gestures or unknown behaviours; the method may furthercomprise the steps of monitoring for unclassified gestures andunclassified behaviours based on repeated patterns, and adding theunclassified gestures and unclassified behaviours to the respective setof gesture definitions and set of behaviour definitions; the method mayfurther comprise the step of comparing at least the operational data toone or more thresholds, and triggering an alarm if one or morethresholds have been exceeded; analysing the operational data toidentify data patterns may comprise comparing data values from thesensors to values in the definitions, convolving signals representativeof the operational data processing signals representative of theoperational data, using machine learning techniques to segment theoperational data, or combinations thereof; the signals may be processedto obtain spatial information, frequency information, time domaininformation, or combinations thereof from the processed signals; datapatterns may comprise first order data structures, gestures comprisesecond order data structures, and behaviours comprise third order datastructures, and the method may further comprise the step of identifyingone or more higher order structure defined by sets of higher orderdefinitions, each higher order structure comprising a combination of twoor more lower order data structures, wherein at least one lower orderdata structure comprises an immediately lower order data structure; anoperational analysis may be generated based on the data structures,which may comprise an efficiency analysis of the duration of each datastructure, and the time between data structures; the operationalanalysis may further comprise an estimated cost of each accomplishmentor activity based on one or more maintenance costs, material costs,labour costs, and equipment cost, and/or a comparison between separateequipment, separate operators, or both separate equipment and separateoperators; the operational analysis may comprise a comparison of theestimated costs and benefits of modified operations relative to theestimated costs and benefits of current operations; the method mayfurther comprise the step of logging each of the one or more gesturesand one or more behaviours in the database.

According to an aspect, there is provided a system for analysing andtracking machine systems, comprising sensors mounted to equipment in themachine system, and a processor in communication with the sensors. Thesensors sense operational data from the equipment comprising at leastlocation, time, and one or more operational condition data related tothe equipment. The processor is programmed to: identify data structuresusing sets of data structure definitions, the data structures beingordered hierarchically in one of a first order and more than one higherorder, wherein the first order data structures comprise data patternsidentified from the operational data, and higher order data structurescomprise an immediately lower order data structure in combination withone or more lower order data structures.

In other aspects, the system described above may further comprise thefollowing aspects, alone or in combination: the system may furthercomprise a notification device, and the processor may be furtherprogrammed to identify potential data structures, and trigger thenotification device to notify a user of any potential data structures;the processor may be further programmed to compare the operational dataor one or more data structures to one or more thresholds, and to triggerthe notification device if one or more thresholds have been exceeded;comparing the data patterns to the database may comprise comparing datavalues from the sensors with values in a database, convolving signalsrepresentative of the operational data with another signal, processingsignals representative of the operational data, applying machinelearning techniques to segment the operational data, or combinationsthereof; the signals may be processed to obtain spatial information,frequency information, time domain information, or combinations thereof.

According to another aspect, there is provided a method of analysing andtracking machine systems, comprising the steps of: sensing operationaldata from equipment, the operational data comprising at least location,time, and one or more operational condition data related to theequipment; and identifying data structures using sets of data structuredefinitions, the data structures being ordered hierarchically, whereinthe first order data structures comprise data patterns identified fromthe operational data, and higher order data structures comprises acombination of two or more lower order data structures, wherein at leastone lower order data structure comprises an immediately lower order datastructure.

In other aspects, the method described above may further comprise thefollowing aspects, alone or in combination: the method may furthercomprise the steps of monitoring for unknown data structures not in thesets of data structures and adding a definition of one or more unknowndata structures to the sets of data structure definitions or alerting auser to classify the unknown data structures; identifying data patternsmay comprise comparing data values from the sensors to values in thedefinitions, convolving signals representative of the operational datawith another signal, or processing signals representative of theoperational data, applying machine learning techniques to segment theoperational data, or combinations thereof; the signals may be processedto obtain spatial information, frequency information, time domaininformation, or combinations thereof from the processed signals; themethod may further comprise the step of generating an operationalanalysis based on a plurality of identified data structures; theoperational analysis may comprise an efficiency analysis of the durationof one or more data structures, and a time interval between selecteddata structures; the operational analysis may further comprise anestimated cost of one or more data structures based on one or moremaintenance costs, material costs, labour costs, and equipment costs;the operational analysis may comprise a comparison between separateequipment, separate operators, or both separate equipment and separateoperators; the operational analysis may comprise a comparison of theestimated costs and benefits of modified operations relative to theestimated costs and benefits of current operations; the method mayfurther comprise the step of logging each of the identified datastructures in a database.

These and other aspects will be apparent from the specification,drawings and claims contained herein. The various aspects may becombined in any reasonable manner as recognized by those skilled in theart.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features will become more apparent from the followingdescription in which reference is made to the appended drawings, thedrawings are for the purpose of illustration only and are not intendedto be in any way limiting, wherein:

FIG. 1 a-1 c is a block diagram of a systemic design for a Single EntityPerception System, showing how the system may work in sensing,abstracting, perceiving, classifying, analysing, and reporting a singleentity's actions.

FIG. 2 is a block diagram of a Multi-Sensor Entity Sensing and LocalEmbedded Computing subsystem, showing the nature of some typical entitysensing for an example where the organization using the system is awaste hauling operation.

FIG. 3 is a block diagram of a Time and/or Spatial Signal Processingsubsystem, showing examples of typical types of signal processing thatmay be applied to signals by the system.

FIG. 4 is a diagram of the Gesture Recognition and Classificationsubsystems, setting out examples of gestural recognition as being basedon perceiving/matching shapes in a multi-dimensional space consisting oftime, three spatial dimensions (“Region” extent) plus particular“constellations” of signals assembled from raw Observations,time/spatially processed signals, and previously recognized Gestures.

DETAILED DESCRIPTION

There is provided a system for automatically sensing, abstracting,perceiving, classifying, analyzing, and reporting regarding the actionsof appropriately instrument-equipped organizational entities inreal-time and/or near-real-time.

Geographically Indifferent Data Acquisition

Referring to FIG. 1 a-1 c , the depicted system, generally indicated byreference numerals 101 to 129, includes electronic sensors and localizedpre-processing computing contained in Multi-Sensor Entity Sensing andLocal Embedded Computing block (101), which sense characteristics fromthe physical world and/or one or more virtual (data simulated/modeled)worlds (100) and convert them into time-stamped, electronicallymediated, measurements (data) of the same. FIG. 2 shows a block diagramof a typical preferred embodiment of sensing and computing block (101)in which an entity (11) is a waste hauling truck, bin, or specialty binsite for the waste industry. These may include a general sensor block(1) in communication with a local processing block (8) and communicationblock (9). Sensor data may then be transmitted through a network (13) toa central computer processor/database, or displayed on a display unit(10), which may also transmit data all or selected data. Sensors mayinclude one or more: RFID reader (2) that communicates with RFID taggedobjects or locations (12), digital camera (3), GPS (4), weight or loadsensor (5), CAN bus (6), etc. Notwithstanding the specific sensorsarticulated in FIG. 2 , it will be clear to one skilled in the art thatadditional sensors (7), capable of sensing dangerous chemicals, density,the type of and volume/weight of specific materials in the waste stream,et al., may also be used, or other sensors required forapplication-specific needs that will be evident to one skilled in theart. Their diversity and capabilities will increase over time as sensortechnologies progress.

Serialization and World Line Creation

Such measurements are conveyed via a Data Transmission block (102) to aSerialization block (103), where measurements from disparate sensors areproperly sorted and time-ordered into the time sequence in which theyoccurred. FIG. 1 shows that certain definitions are user defined, asindicated by the block “humans” providing input to various blocks.

These, now properly time-ordered separate sensor data streams are thenfed to Entity 4D World Line Record Creation block (104). Here they areordered into data set records that specifically associate the entity's4D location in time and space (3 spatial dimensions plus time) with themeasurement taken. Geographic 3D position information comes fromspecific position determining sensors such as for example, a GPSreceiver module. The term “world line” is used in this document in thesense of a physics “world line” i.e. the trajectory that an object takessimultaneously through 4 dimensional time and space. The world line ofeach entity is tracked by the system and (later) marked up withperception annotations that characterize “Regions” (of time and space)along the world line associated with identified/classified actions thatoccurred within said Regions.

These records are then stored into a 4D World Line Observations Database(105) in a form allowing the time/space location links to be associated,stored, and retrieved with each sensor observation. In a preferredembodiment, a “NoSQL” database such as MongoDB may be used to enableconstruction of particular “tree” and “forest” data structures ofrelated measurements and higher level abstracted/perceivedobservation-based information, but other database types are possible andevident to one skilled in the art.

Time and/or Spatial Signal Processing

The system's preferred architecture is a real-time one commonly known as“data flow”. Incoming data records are stored into the database forlater reference, but are subsequently immediately pulled and processedby Gesture Recognition and Classification block (109), passing throughTime and/or Spatial Signal Processing block (107) in the process. Thesetwo processing subsystems (107 and 109) are complimentary. FIG. 3 showsa typical structure of Spatial Signal Processing block (107).Measurements flowing from the 4D World Line Observations Database (FIG.1 , (105)) are routed by a Sensor Signal Router (20) to the appropriatesignal processing block(s), both routing and processing parameters beingdetermined from the Signal Processing Definitions block (106) of FIG. 1. Based on these settings, sensor specific signal analysis is provided,generating an array of additional Processed Signal (29) information fromvarious blocks representing various operations, such as convolutions(21), auto-correlations (22), comb or multi-tap filters (23), FourierTransforms (24) wavelet transforms (25), digital frequency filters (26),time and/or geographic window assembly (27), and/or other signalprocessing algorithms (28). This Processed Signal (29) and Raw Signal(30) information is made available to the Gesture Recognition andClassification subsystem (109) shown in FIG. 1 and subsequentsubsystems, where it is used in addition to the raw observational datato make determinations regarding the nature of perceived entity actions.

The method and system may be useful to process such signals in more thanone dimension. Since the data being fed into the signal processing isboth time based and spatially based, it is possible and intended thatthe nature of processing may include—jointly and severally—any/allcombination(s) of the 3 spatial dimensions plus time, plus the sensorreadings themselves.

The specific signal processing blocks depicted in FIG. 3 are exemplaryonly. Depending on the nature of the entity and the actions beingperceived by the system, other signal processingmethods/algorithms/techniques may be used and will be evident in contextto one skilled in the art.

The method or system may also be used to provide the ability to dealwith the reality that time-based measurements are continuously flowing.Analysis, pattern recognition, and entity featureidentification/perception based on such continuous flows is differentfrom, for example, machine vision analysis of a single photograph,wherein all data relevant to the features being perceived is certain tobe contained. We refer to this as the “Picture Windowing Problem”.

For example, in one embodiment, signal processing subsystem Time and/orGeographic Window Assembly (27) may be used. This subsystem compositessensor readings into non-time-continuous windows, effectively creating adata stream consisting of several different “tap points” in time and/orspace, offset to one another. In cases where known delay relationshipsbetween signals have been established, this composite data flow is mucheasier to analyse and will inherently highlight associated inter-signalrelationships. Since the world line is inherently a 4 dimensional space,said compositing and setting of tap points may occur across any/all ofthe 3 spatial dimensions and/or time.

Data Structure Recognition and Classification

Once they have passed through the signal processing subsystem, worldline sensor data is fed to Gesture Recognition and Classification block(109). It is here that further processing of the sensor signals occurs.FIG. 4 shows a conceptual view of this subsystem.

Each entity action, such as for example (in a preferred embodiment), awaste truck bin lift, has a particular time/space data flux“fingerprint”. Transitions between these can be recognized using machinelearning data discretization techniques. Having segmented suchtransitions, the body of data between transitions can then be analysedfor maximum likelihood indicators that then can be automatically set asdefinitions of such actions.

For simplicity, in FIG. 4 , the 3 spatial dimensions are compressed ontoa single axis labelled “Space/Region”, but it should be understood thatthis single axis actually represents 3 separate spatial axes ofdimensional state space. The other part of this method and system'ssolution to the Picture Windowing Problem is found herein. For example,a Gesture Instance Builder subsystem (40) may work with GestureDefinitions block (108) to create particular software object instancestuned specifically to look for particular gestures. Conceptually, theseobjects are somewhat like immune system cells—they search through thestream of time, space, and multiple sensor data readings, looking tomatch particular patterns of sensor signals, 3D spatial positions,temporal positions, and 4D time/space/signal values of previouslyrecognized Gestures. When a definition match is found, a recognizedgesture is linked with the matching sensor values and time/space datainto a data tree structure using World Line Time/Space RegionAuto-Segmentation block (122) of FIG. 1 . In FIG. 4 , this isrepresented by Gesture A (41), Gesture B (42), Gesture C (43), andGesture D (44), each of which includes certain datapoints as part of arecognizable pattern, which may be made up of raw data, processed data,or a combination of both. As part of its content, such a structureidentifies and defines the “Region” of time/space extent occupied by theGesture. Recognized Gestures are also given a classification type, sucha type being useful in understanding the nature of entity actionsoccurring and tabulating reporting regarding the aggregate of manyactions of the same or similar/related type. Such typing may beexplicitly defined as part of the Gesture's definition. However, it mayalso be the case that, while a definition exists for a uniquelyclassified/typed Gesture, its type name and/or the type's relation toother known types is not presently known. Such a case may, for example,arise when an automatically generated Gesture definition is matched. Insuch a case, unless the system's human operators have explicitly entereda type classification and specific type relationship metadata to modifythe automatic definition, all the automated system knows is that this isa uniquely recognizable Gesture, different from other Gestures (seebelow). Such un-typed Gestures may be flagged by the system in New KnownGesture block (46) for human operators to intervene and use theirknowledge of the context of operations surrounding the gesture to defineits type and provide additional type metadata that allows this Gestureto henceforth be correctly named and tabulated into reporting summariesusing the type and the type metadata defining the type's relationship toother types.

The Gesture's structure is also published to be available for assistingin recognition of other gestures and, as well, is stored in a databaseusing Entity World Line Markup State Set Database block (122) shown inFIG. 1 .

This database contains a description of the “State Set” of an entity asit traverses its 4D time/space world line. The world line markupindicates the perceived/recognized actions performed by the entity, andthe Regions of time/space over which they occurred. As such it can beanalysed to generate analytic summaries of its records, allowingcreation of summaries of what types of actions happened, the extent oftime and space over which they happened, and, as well, metadataregarding the relationships between perceived/recognized actions ofvarying levels of abstraction. Such analysis is performed by Entity FactAnalytics (127) and Gesture, Behaviour, Activity, AccomplishmentAnalytics/Reporting block (128), and is discussed later in more detail.

Such “gesture trees” create a de facto custom window into themulti-dimensional data, and allow for the creation of other windowsaround their state space location that can be used by any other gestureinstance recognizer to effectively centre its window onto the previouslyrecognized gesture. In this way, the individual “gesture trees” mayengender other recognized gestures, eventually forming a sort of“gesture forest” data set representing recognized gestures within themulti-dimensional state space.

It may be the case—especially initially—that the system does notrecognize any Gestures. In this case, the unclassified/unrecognized fluxof sensor and time/space data is fed to Automated Data-Driven GestureClassification and Definition block (45) shown in FIG. 4 . Thissubsystem examines the data flux, attempting to identify points at whichactions change, signalling a transition from one action to another,different one. The nature of such classification algorithms have beenthe subject of so-called “machine learning” research, the outcome ofwhich has been a variety of techniques for what is sometimes termed“data discretization”—effectively detecting transition points betweenone data context and another. When concerning temporal data, suchtechniques divide into two main categories—“supervised” (where thenature/context of such transitions is understood a priori) and“unsupervised” (where there is no real context to assist in identifyingtransitions). Such techniques are known to one skilled in the art, and avariety of them may be employed as part of an embodiment. Examples ofsome possible techniques are contained in the paper “Discretization ofTemporal Data: A Survey” by P. Chaudhari, R. G. Mehta, N. J. Mistry, andM. M. Raghuwanshi, but others, equally or more applicable, will beevident to one skilled in the art. Application of such techniquesidentifies transition points, where it is probable that thepost-transition action occurring is different from what was occurringimmediately previous to the transition. These, in turn, allow analysisof the two different data fluxes (pre and post transition) to determinemaximum likelihood indicators for identifying future actions of asimilar nature and, as well, for uniquely identifying the action by wayof a particular “fingerprint” of time/space related data values frommultiple sensors. This, in turn, gives rise to automated creation of aGesture definition template for identifying future occurrences of thisGesture and classifying them into the same Gesture type category.

The method and system may be used to provision tiered perception andrecognition of successively higher level abstractions of actions basedon multi-dimensional recognition of either human-defined ormachine-generated action definitions (e.g. Gestures). Thus the methodand system, as described, allows fundamental Observations to beabstracted to perceive Gestures; Gestures plus Observations to becombined to abstract and perceive/recognize higher level “Behaviours”;Behaviours plus Gestures plus Observations to be abstracted toperceive/recognize yet higher level “Activities”; and Activities plusBehaviours plus Gestures plus Observations to be combined toabstract/perceive/recognize yet higher level “Achievements”. While 5levels of abstraction are articulated in this description, there is noreason that such a process of abstraction—based as it is upon acombination of all raw Observations plus all previously perceived lowerlevel and current level actions—cannot extend to yet higher levels.Generalization of such a process to higher levels will be obvious to oneskilled in the art.

Given this tiering of perception/recognition, the functioning of thesuccessive levels of perception/recognition is similar to that of thefirst level Gestures with respect to: perception/recognition algorithms(109), definitions (108), and automated data-driven classification andautomated definition (110) for the higher level abstractions, or higherorder data structures—Behaviours, Activities, and Accomplishments. Theonly difference is that, for each successive level of abstractedperception, more information is available to informclassification/perception/recognition choices, as all previouslyperceived/recognized lower or current level actions are available inaddition to the raw Observation data itself. Once the utility ofsuccessive levels of abstraction is appreciated in conjunction with thelower level approach to action-centric “windowing” of data and matchingof definitions, creation of higher levels or orders of data structuresshould be evident to one skilled in the art. Thus the FIG. 1 blocks(112) through (120) inclusive, which include a definitions database(112, 115, 118), an Automated data driven classification and definitionblock (113, 116, 119), and a recognition and classification block (114,117, 120) do not need further description.

World Line Time/Space/Region Auto-Segmentation

“Regions” of extent in time and space may be identified within whichactions occur. As an example, Region Definitions block (121) in FIG. 1is a repository of definitions of such regions. Regions can be createdexplicitly by human operators and entered into this repository. In apreferred embodiment applied to a waste hauling company, human operatorsmight, for example, define regions of interest such as a truck depotyard, land fill, or large area client site as Regions for whichknowledge of entity presence/absence was desirable. These explicitdefinitions would be stored internally as object classes, with theirgeographic extent defined, but with an undefined time extent. When thisdefinition was matched up using World Line Time/Space RegionAuto-Segmentation block (122), a specific instance variable of thatRegion would be created with the time extent filled in. This would thenbe attached to the action presently being perceived, and stored as partof the entity's world line markup information in Entity World LineMarkup State Set Database (123).

The method and system is preferably able to identify and classify bothentities and “associated entities” automatically through their sensordata fluxes, perceived actions, and time/space relationships betweensaid actions. An associated entity is an additional entity that isconnected in some manner with another one. For example, in a preferredembodiment applied to a waste hauling organization, a truck could be anentity, and the truck's driver would be an associated entity connectedto the truck for some temporal period.

Such identification/recognition of an entity, such as a truck and anassociated person driving the truck, may be accomplished using so-calledmachine learning techniques in a manner similar to that described withrespect to Automated Data-Driven Gesture Classification and Definition.As with entity action perception/recognition/classification in theidentity recognition and classification block (124), each entity such as(in a preferred embodiment) a truck has a particular “fingerprint” ofsensor data, system perceived actions, and metadata surroundingrelations between actions, which may be defined or stored in identitydefinitions block (111). Transitions between entities and/or associatedentities (such as, for example, a truck's driver) can be recognizedusing machine learning data discretization techniques. Having segmentedsuch transitions, the body of data between them can then be analysed formaximum likelihood indicators that can act as definitions of suchentities' presence during particular temporal time periods. Suchdefinitions are stored in Entity and Associated Entity (Operator)Definitions (125). Thus activity of particular entities and associatedentities can be automatically recognized repeatedly by the system. Humanoperators of the system can edit these definitions, adding in metadatasuch as names, truck VIN numbers, etc. to provide more specificcontextual identification. Once this metadata has been added, it can bestored as a more complete identity element of the Entity World LineMarkup State Set Database (123), where it can be made available to theEntity fact analytics calculations (126), Entity Fact Database (127),Gesture, Behaviour, Activity, and Accomplishment Analytics/Reportingblock (128) used in compiling analytics and reporting information.

Action-Based Analytics, Reporting, and Billing

The nature of the analytics provided can satisfy multiple organizationalassessment, optimization, and strategy goals. The Entity World LineMarkup State Set Database (123) contains automatically perceivedinformation about the actions performed by the entity over time. At avery basic level, such information allows construction of a “fact”database that tallies common figure of merit performance statistics overuseful periods of time such as per day, per week, month, year, etc. In apreferred embodiment applied to a waste hauling organization, thesemight be, for example, daily/weekly/monthly facts about how many wastebins were emptied, what the average bin lift time was, how much truckidling existed, how much fuel was consumed over the 3D terrain pathdriven, or as perception events occurring in the course of a day, week,month, or year.

Beyond such basic operations summary performance tallies, however, moresophisticated analysis leading to real-time or near-real-timeoptimization can also be performed:

Cyclical temporal analysis may be performed to detect and understandboth normal action levels and deviations therefrom. Actions can beaggregated over multiple continuous time periods such asdays/weeks/months, etc. They can also be examined over specificallynon-continuous segments, such as looking at all Mondays compared to allThursdays, summer compared to winter, etc. As well, they can beaggregated geographically before such temporal analysis, for examplebeing grouped regarding specific geographic regions identified by thesystem.

Such time/space aggregations of action data can then be analysed interms of frequency distribution, statistical measures such as standarddeviation that measure the variance of actions of the same or similarnature, cause/effect relationships regarding modulation of duration ofactions, or other analytic analysis evident to one skilled in the art.These summaries may be compared with historic averages over the sametime intervals, thereby establishing statistical variances of thesemeasures over multiple time cycles. Such comparisons and variancemeasures may then be further analyzed to identify and flag statisticallysignificant deviations for human investigation/optimization/remedy.

Analysis may also be non-temporal, using frequency analysis,auto-correlation, wavelet transforms, and/or other signal processingtechniques similar to those detailed in FIG. 3. Time and/or SpatialSignal Processing to detect performance patterns.

As well, such overall entity performance analysis may be based onmachine learning algorithm approaches similar to those already detailedfor entity action recognition/classification, allowing automaticsegmenting/classification of entity performance, development of maximumlikelihood estimators to identify each classification type, andanalysis/establishment of cause/effect relationships between variables.This automatic elucidation of the structure of each entity's performanceand creation of cause/effect understanding of the causes of suchstructure is a significant advance over present day organizationalanalysis capabilities.

Automatic Detection of Performance Deviations from Historic Functioning

Taken together, these multiple analysis types enable significantmanagement optimization opportunities: Firstly, they enable generationof “Exception Events” in real-time or near-real-time, where it is clearthat something unusual has happened to the entity out of the realm ofnormally expected daily occurrence. A simply example of these events, ina preferred embodiment applied to a waste hauling organization, would beif a truck suddenly became idle for more than a certain period of time.Such inaction would be perceived by the system, identified as a “truckidle” exception event, and reported immediately to dispatch operators.Secondly, more subtle deviations could also be perceived, allowing oneto assess the slow changing of an entity's performance functionalityover time and/or in response to operational changes implemented. Forexample, in a preferred waste hauling embodiment, a truck's power takeoff (“PTO”) unit, sensed via truck CAN bus data fluxes, might slowlydegrade in terms of power delivery over time due to equipment wear. Thiscould cause a lengthening of the lift time of so-called “Roll Off” wastebins onto the back of the truck, which would be noted in performancemetrics. Such a performance degradation could be identified andmeasured, then correlated with the CAN bus PTO data by the system'smachine learning segmentation techniques to establish a probable causalrelationship between the two, which could, in turn, be identified tohuman operators.

Assessment of Separate Categories of Actions and Derivation of OverallPer Entity Efficiency

When entity actions are classified by type, they can be tabulated bytype over known periods of time and/or space. It is an aspect of anembodiment that such types can also be given “attributes” by humanoperators who understand the greater context of operations. Thus typesof actions can be sorted and tabulated by attribute. For example, in apreferred embodiment where the system is applied to a waste haulingbusiness, revenue-generating actions such as waste bin pickups fromclients might be given a “productive time” attribute, whereasrevenue-costing actions such as time spent at a landfill, time spentidle, etc. might be given an “unproductive time” attribute. Performanceof an entity could be evaluated over a specific time period to examineits entity-specific ratio of productive to unproductive time, allowinggeneration of a measure of its efficiency. Such entity-specificefficiency figures could then be compared to cross-fleet averages to,for example, identify outlier entities whose performance needed humaninvestigation and/or correction.

Assessment of Per Entity Profit/Loss/Cost

Related to such efficiency analysis, it is an additional aspect of anembodiment to enable per entity assessment of profit, loss, and cost andthe correlation of these values with the entity state set informationstored in the Entity Fact Database (127) and Entity World Line MarkupState Set Database (123) to understand cause/effect relationshipsbetween the automatically perceived actions/regions and theirprofit/loss/cost outcomes. Based on such analysis, deep understanding ofthe incremental cost and profit/losses arising from adding/subtractingparticular actions can be obtained, allowing optimization of chains ofactions to maximize profitable outcomes. For example, in a preferredembodiment where the organization was a waste hauling company, it wouldbe possible to assess the specific incremental “transition cost” ofadding one customer's pick up to a particular route, measuring theincremental time taken to pick up, and separating out the incrementaleffect of this waste pickup on when a trip to dump at a landfill wasneeded. This sort of entity-specific, action-specific, client-specific,cost calculation is not presently possible. It is invaluable indetermining cost/benefit, assessing pricing and opportunity cost forcurrent or future clients, and for optimizing routing of trucks basednot only on geography, but on the nature of what they have historicallypicked up from specific locations in terms of weight, volume, material,etc.

Aggregation of Multiple Entities into Groups of Similar Auto-ClassifiedType

While much of this discussion is focused upon automatic perception andmeasurement of actions per entity, it will be obvious to one skilled inthe art that such entity measurements can be usefully combined, grouped,and aggregated. This is particularly the case given the method andsystem's ability to automatically classify types of actions, and formetadata regarding relationships between those actions to be eitherautomatically generated by the system, or entered directly by humansfamiliar with action contexts who are able to define and name saidaction types and their relations to each other. Thus it is possible forthe system to generate reporting that groups entities by type, and,further, analyses based on more sophisticated metadata such as causalrelationships between types of actions, etc.

Comparison Across Multiple Entities with Varying Associated Entities, orRegarding a Single Associated Entity Over Time

It should also be evident to one skilled in the art that it is possibleto generate reporting that directly compares or ranks associatedentities such as, for example, operators of vehicles. Since the methodand system can classify—through the nature of, and relationshipsbetween, their actions—which human was operating the entity, it ispossible to generate inter-human rankings of groups/teams of operatorsregarding their operation (at different times) of the same entity.Additionally, it is possible to generate similar inter-human rankings ofoperators and their operation of other entities of a similar type (forexample, multiple trucks of the same model/type). As well, it ispossible to assess performance of a single operator over time to measureskills improvement.

Comparison across Multiple Entities and/or Groups of Entities

It should also be evident to one skilled in the art that it is possibleto aggregate and compare actions and automatically analysed/reportedperformance of multiple entities. This is particularly useful incomparing similar, or related, entities and examining potentialcause/effect relationships for significant differences between them. Forexample, in a preferred embodiment applied to a waste haulingorganization, it might be the case that truck engine wear for one set oftrucks used in a particular geographic terrain was significantly worsethan that of the same trucks used elsewhere. Similarly, waste bins couldbe assessed to establish causal factors with respect to their effective(non-chronological) age and repair status versus client, location,weight of materials, local rainfall levels, etc. Once established, suchcausal modelling could be used predictively to anticipate and/ormitigate entity maintenance activities/costs.

Automated Action-Based Billing

It is a further aspect of the system and method that it enablesAutomated action-based billing block (129) to generate customer chargesbased on specific, automatically perceived and tabulated, actual actionsand achievements completed rather than broad contractual agreements.Using the system and method, it is possible to automatically perceivedcompleted, billable, accomplishments and, in detail, determine the costsof the accomplishments. Such detailed reporting may be used toautomatically generate billing, particularly “cost plus” billing thatensures a known profit margin per action.

For example, in a preferred embodiment such as application to a wastehauling organization, it would be possible to automaticallytabulate—over an arbitrary billing period or even on a per eventbasis—the number of times a specific truck/driver had gone to a client'ssite and picked up a waste bin. It would further be possible, using themetadata attached to each system-perceived action, to base thataccomplishment's billing on a very detailed number of action-relatedvariables such as: the weight of material picked up by the truck eachtime; the incremental transit time and fuel consumption both from thetruck's previous location to the pickup site and to a landfill fordumping; and the indirect cost of truck wear and tear for carrying sucha weight of waste material.

Based on this specific, per event, information, costs can be determined.Billing can then be generated on a per event basis for thisaccomplishment, reflecting actual accomplishment costs plus a desiredprofit margin. Alternatively, billing can be based on simpler, butequally automatically perceived, accomplishments such as just lifting abin at a particular site. However, in both cases, billing is generatedonly when the event actually happens and is not based on a contract thatcalls for emptying bins on a call-in basis, “on average every twoweeks”, etc.

Such evidence-based, action-based, billing is extremely powerful interms of both strategic and tactical management of the organization. Itconfers ability to directly manage and optimize the organization on aper entity and per action profit/loss/cost basis. This capability isspecifically enabled by the ability to automatically perceive, record,and aggregate detailed information about each action.

Automated Assessment of Performance Response to a Known Recipe ofOperations Changes

The system and method may also be used to enable automatic assessment ofthe effect of a known set of operational changes—both per entity, andwith respect to groupings of entities. The significant per entity levelof detail perceived by the system regarding entity actions allowsperformance metrics to be evaluated both before and after changes aremade. Thus the system and method can analyse the response of theorganization to changes, essentially treating it in a manner similar toan electronic filter and assessing its “impulse response” to aparticular type of stimulation. Such response assessment can happen innear-real-time, waiting only on the individual time constants that maybe associated with the specific recipe of changes implemented. It isimportant to note that such a response is not necessarily linear—eitherper entity or across all system-recognized entities or entity groups.Without the ability to automatically perceive and measure real-time, perentity, actions, and assess them against continuously changing historicnorms, such response assessment would be impossible. It is the finenessof real-time-automated, per entity, time/space/action perception thatmakes such response assessment possible/viable.

In this patent document, the word “comprising” is used in itsnon-limiting sense to mean that items following the word are included,but items not specifically mentioned are not excluded. A reference to anelement by the indefinite article “a” does not exclude the possibilitythat more than one of the element is present, unless the context clearlyrequires that there be one and only one of the elements.

The following claims are to be understood to include what isspecifically illustrated and described above, what is conceptuallyequivalent, and what can be obviously substituted. The scope of theclaims should not be limited by the preferred embodiments set forth inthe examples, but should be given the broadest interpretation consistentwith the description as a whole.

What is claimed is:
 1. A method of analyzing and tracking a plurality of waste hauling trucks in a geographic area, wherein each waste hauling truck of the plurality of waste hauling trucks includes a sensor mounted to peripheral equipment, the method comprising: receiving operational data from the peripheral equipment of each waste hauling truck in the plurality of waste hauling trucks via the sensor, the operational data includes: a location measured by a GPS receiver, a time, and sensor data related to operation of the peripheral equipment of each waste hauling truck; analyzing the operational data to identify a data pattern of the plurality of waste hauling trucks; storing the data pattern in a database; classifying the data pattern as a gesture based on a set of gesture definitions with a first machine learning technique; identifying a system behaviour based on a plurality of gestures, wherein the plurality of gestures includes the gesture, and wherein the system behaviour is identified with a second machine learning technique, wherein the second machine learning technique is different from the first machine learning technique; and generating a report, the report to associate the system behaviour of the plurality of waste hauling trucks to determine an efficiency of an aggregated operation of the plurality of waste hauling trucks across the geographic area, wherein analyzing the operational data to identify the data pattern comprises: comparing the operational data to values in the set of gesture definitions, processing raw signals representative of the operational data to generate processed signals, and further processing the processed signals to identify the data pattern.
 2. The method of claim 1, further comprising monitoring for an unknown gesture and adding the unknown gesture to the set of gesture definitions.
 3. The method of claim 2, further comprising alerting a user to classify the data pattern when the data pattern is unable to be classified in any gesture definition of the set of gesture definitions.
 4. The method of claim 1, further comprising triggering an alarm if the sensor data exceeds a threshold value.
 5. The method of claim 1, wherein the the processed signals include spatial information, frequency information, time domain information, or combinations thereof from the data pattern.
 6. The method of claim 1, wherein the data pattern comprises first order data structures, the gesture comprises second order data structures, and the system behaviour comprises third order data structures, and further comprising identifying a higher order structure defined by sets of higher order definitions, each higher order structure comprising a combination of two or more lower order data structures, wherein at least one lower order data structure comprises an immediately lower order data structure.
 7. The method of claim 6, further comprising generating an operational analysis based on the data structures.
 8. The method of claim 7, wherein the operational analysis comprises an efficiency analysis of a duration of each data structure, and the time between data structures to determine the efficiency.
 9. The method of claim 8, wherein the operational analysis further comprises an estimated cost of an accomplishment or an activity based on one or more of maintenance costs, material costs, labour costs, and equipment costs.
 10. The method of claim 9, wherein the operational analysis comprises a comparison between separate equipment, separate operators, or both separate equipment and separate operators.
 11. The method of claim 9, wherein the operational analysis comprises a comparison of estimated costs and benefits of modified operations relative to the estimated costs and benefits of current operations.
 12. A system for analyzing and tracking a plurality of waste hauling trucks in a geographic area, the system comprising: sensors mounted to peripheral equipment in each waste hauling truck of the plurality of waste hauling trucks, the sensors sensing operational data from the peripheral equipment comprising: a location measured by a GPS receiver, a time, and one or more additional operational condition data related to operation of the peripheral equipment; and a processor in communication with the sensors of each waste hauling truck of the plurality of waste hauling trucks, the processor being programmed to: identify data structures using sets of data structure definitions, the data structures being ordered hierarchically in a first order data structure with a first machine learning technique and a second order data structure with a second machine learning technique, wherein the first order data structure comprises data patterns identified from the operational data, and the second order data structure comprises the first order data structure in combination with an additional first order data structures to determine an efficiency of an aggregated operation of the plurality of waste hauling trucks across the geographic area.
 13. The system of claim 12, further comprising a notification device to notify a user of if one or more thresholds have been exceeded in the operational data.
 14. The system of claim 12, further comprising comparing the data patterns to values in a database, processing raw signals representative of the operational data with another signal to generate processed signals, processing the processed signals representative of the operational data, applying machine learning techniques to segment the operational data, or combinations thereof.
 15. The system of claim 14, wherein the processed signals include spatial information, frequency information, time domain information, or combinations thereof.
 16. A method of analyzing and tracking a plurality of waste hauling trucks in a geographic area, wherein each waste hauling truck has peripheral equipment, the method comprising: receiving operational data from the peripheral equipment of each waste hauling truck of the plurality of waste hauling trucks, the operational data comprising: a location measured by a GPS receiver, a time, and sensor data related to operation of the peripheral equipment; identifying a data pattern from the operational data of the plurality of waste hauling trucks; identifying data structures from the data pattern using sets of data structure definitions, the data structures being ordered hierarchically, wherein first order data structures include the data pattern, and higher order data structures comprises a combination of two or more lower order data structures, wherein at least one lower order data structure comprises an immediately lower order data structure, and wherein a first machine learning technique is used to identify the first order data structures and a second machine learning technique is used to identify the higher order data structures to describe system behaviours; and generating an operational analysis based on a plurality of identified data structures from the plurality of waste hauling trucks across the geographic area, wherein the operational analysis comprises an efficiency analysis of a duration of one or more data structures, and a time interval between selected data structures, and wherein the operational analysis further comprises an estimated cost of the one or more data structures based on one or more maintenance costs, material costs, labour costs, and equipment costs, wherein analyzing the operational data to identify the data pattern comprises: comparing the operational data to values in a set of gesture definitions, processing raw signals representative of the operational data to generate processed signals, processing the processed signals to identify the data pattern, and using machine learning techniques to classify the data pattern as a gesture based on the set of gesture definitions, and wherein the operational analysis is to optimize a routing of the plurality of waste hauling trucks within the geographic area.
 17. The method of claim 16, wherein the raw signals are processed to obtain processed signals, wherein the processed signals include spatial information, frequency information, time domain information, or combinations thereof from the data pattern.
 18. The method of claim 16, wherein the operational analysis comprises a comparison between separate equipment, separate operators, or both the separate equipment and the separate operators.
 19. The method of claim 16, wherein the operational analysis comprises a comparison of estimated costs and benefits of modified operations relative to the estimated costs and benefits of current operations.
 20. The method of claim 1, wherein the peripheral equipment is a bin lift.
 21. The method of claim 20, wherein the sensor data is collected from a load sensor to measure a weight of a bin lifted by the bin lift.
 22. The system of claim 12, wherein the peripheral equipment is a bin lift.
 23. The system of claim 22, further comprising a load sensor to measure a weight of a bin lifted by the bin lift.
 24. The method of claim 16, wherein the peripheral equipment is a bin lift.
 25. The method of claim 24, wherein the sensor data is collected from a load sensor to measure a weight of a bin lifted by the bin lift.
 26. The method of claim 1, further comprising optimizing a routing of each waste hauling truck of the plurality of waste hauling trucks based on the efficiency of the operation of the plurality of waste hauling trucks across the geographic area.
 27. The system of claim 12, wherein the processor is programmed to optimize a routing of the waste hauling truck based on the efficiency of the operation of the plurality of waste hauling trucks across the geographic area.
 28. The method of claim 5, wherein the processed signals are processed with one of convolutions, auto-correlations, comb or multi-tap filters, Fourier Transforms, wavelet transforms, digital frequency filters, time and geographic window assembly.
 29. The method of claim 28, wherein the processed signals are processed with the raw signals with the second machine learning technique.
 30. The method of claim 1, further comprising automatically generating a bill based on the report. 